Analysis of Multi-Criteria Metrics for Classifier and Algorithm Evaluation

Evaluation of classifiers and supervised learning algorithms has historically been done by estimating predictive accuracy via cross-validation tests or similar methods. More recently, ROC analysis has been shown to be a good alternative. However, the characteristics can vary greatly for different domains and it has been shown that different evaluation methods are appropriate for different problems. Moreover, for many problems, methods that combine two or more criteria have been successful. We introduce the software engineering concepts of quality attributes and metrics to shift focus from the question of which evaluation method should be used to which quality attributes need to be assessed for a particular application. We analyze a large number of metrics and categorize them according to which attribute(s) they address. Finally, a structured way to evaluate classifiers and supervised learning algorithms is proposed, as well as a generic multi-criteria-based metric that can be used to combine arbitrary metrics.